Systems And Methods For Detecting And Dynamically Mitigating Driver Fatigue

ABSTRACT

This technology relates to dynamically detecting, managing and mitigating driver fatigue in autonomous systems. For instance, interactions of a driver in a vehicle may be monitored to determine a distance or time when primary tasks associated with operation of the vehicle or secondary tasks issued by the vehicle computing were last performed. If primary tasks or secondary tasks are not performed within given distance thresholds or time limits, then one or more secondary tasks are initiated by the computing device of the vehicle. In another instance, potential driver fatigue, driver distraction or overreliance on an automated driving system is detected based on gaze direction or pattern of a driver. For example, a detected gaze direction or pattern may be compared to an expected gaze direction or pattern given the surrounding environment in a vicinity of the vehicle

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No. 16/226,322filed Dec. 19, 2018, the entire disclosure of which is incorporated byreference herein.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers or items fromone location to another. Such vehicles may operate in a fully autonomousmode where passengers may provide some initial input, such as a pickupor destination location, and the vehicle maneuvers itself to thatlocation. However, in some situations a human driver may be locatedwithin the compartment of the vehicle. In such situations the driver mayneed to assist in operation of the vehicle or assist a passenger in someway (“assist driver”). With regard to the former, this may occur, forexample, during test runs of a vehicle. With regard to the latter, suchsituations may occur where, for example, a passenger has a problem or isconcerned or otherwise uncomfortable. Here, a human driver in thevehicle may communicate with the passenger and reassure him or her thatthe vehicle is functioning normally, check on the passenger during theride, and/or provide instructions to the passenger in the event of aproblem with the vehicle or an emergency. As the assist driver may notalways be cognitively engaged during a drive, he or she may becomefatigued, engage in distracting tasks or possibly over-rely onautomation. And while the foregoing has been discussed in respect ofautonomous vehicles, such fatigue may also become a factor for driversoperating non-autonomous vehicles.

BRIEF SUMMARY

One aspect of the disclosed technology provides a dynamic interactivecognitive task method for a self-driving system. The method comprisesdetermining, by one or more processors of the self-driving system, aprimary task demand metric associated with a primary task; andcomparing, by the one or more processors, the primary task demand metricto a threshold value. The method may also comprise triggering, by theone or more processors, the self-driving system to initiate aninteractive cognitive task to a driver of the self-driving system basedon a result of the comparison.

In this aspect of the technology, the primary task demand metriccomprises a value associated with one of a comment by the driver, acomment rating, or a request for use of the self-driving system for apassenger of the self-driving system. In addition, as an example, thethreshold value comprises a time threshold value of at least 10 minutes.The threshold value may also comprise a distance threshold value of atleast 10 miles.

Further, triggering the interactive cognitive task comprises initiatinga task that requires a response from the driver. In addition, initiatinga task that requires the response may comprise initiating a task thatrequires use of a motor skill and a cognitive skill as part of theresponse from the driver.

As a further example, triggering the interactive cognitive task maycomprise generating a voice request for the driver to complete anon-visual button push task.

The method may also further comprise monitoring, by the self-drivingsystem, for a response from the driver to the initiated interactivecognitive task and issuing a follow-up interactive cognitive response ifthere is no response within a predetermined time limit.

In additional examples, the method may further comprise determiningcomplexity of the interactive cognitive task based on at least acomplexity of the primary event. Furthermore, a frequency associatedwith triggering the self-driving system to initiate the interactivecognitive task is adjustable based on the result of the comparison. Themethod may also further comprise cancelling the interactive cognitivetask if another primary task event is detected by the self-drivingsystem within a time limit. In addition, the primary task demand metricmay comprise at least one of a time travelled by the self-driving systemor a distance traveled by the self-driving system.

In another aspect, the technology provides dynamic interactive cognitivetask method for a self-driving system, comprising: monitoring, by one ormore processors of the self-driving system, a primary task demand metricassociated with at least one primary task; comparing, by the one or moreprocessors, the monitored primary task demand metric to a threshold; andinitiating, by the one or more processors, using the self-driving systeman interactive cognitive task based on a result of the comparison.

Further in accordance with this aspect of the technology, the primarytask demand metric comprises a distance travelled since performance of aprimary task demand event by a driver of the self-driving system.Additionally, the primary task demand metric comprises an elapsed timesince performance of a primary task demand event by a driver of theself-driving system. The primary task demand metric may also compriseone of a comment by the driver, a comment rating, or a request for useof the self-driving system for a passenger.

In another aspect, the technology comprises a method for managing driverinattention. The method may comprise acquiring, by one or moreprocessors, images of one or both of a driver's eyes to determine anactual gaze direction or pattern of the driver; determining, by the oneor more processors, an expected gaze direction or pattern of the driverbased on a planned route of the vehicle and objects in an externalenvironment of the vehicle; and comparing, by the one or moreprocessors, the actual gaze direction or pattern to the expected gazedirection or pattern. The method also comprises taking corrective actionupon determining that the driver's visual scanning of the drivingenvironment indicates driver inattention.

Further in accordance with this aspect of the technology, determiningwhether the driver's visual scanning of the driving environment deviatesfrom the expected gaze direction or pattern may comprise using anormative model based on environmental statistics to determine theexpected gaze direction or pattern given a planned driving path.Determining whether the driver's visual scanning of the drivingenvironment deviates from the expected gaze direction or pattern mayalso comprise using a normative model based on human visual behaviorstatistics to determine the expected gaze direction or pattern.

Further still, acquiring images may comprise monitoring one or both ofthe driver's eyes for eye closure. In addition, the driver may comprisea driver of a non-autonomous vehicle or an assist driver monitoring theoperations of an autonomous vehicle

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of an example vehicle in accordance withaspects of the disclosure.

FIG. 2 is an example external view of a vehicle in accordance withaspects of the disclosure.

FIGS. 3A and 3B are examples of internal views of vehicles in accordancewith aspects of the disclosure.

FIG. 4 is an example pictorial diagram of a system in accordance withaspects of the disclosure.

FIG. 5 is an example functional diagram of a system in accordance withaspects of the disclosure.

FIG. 6 is an example flow diagram in accordance with aspects of thedisclosure.

FIGS. 7A and 7B is an example flow diagram in accordance with aspects ofthe disclosure.

FIG. 8 is an example flow diagram in accordance with aspects of thedisclosure.

DETAILED DESCRIPTION Overview

The technology relates to detecting, managing and mitigating driverfatigue and distraction in autonomous systems. Some autonomous systemsrequire human oversight. For example, in a vehicle capable of partiallyor fully autonomous driving or self-driving system (SDS), an assist orsafety driver may ride in the compartment of the vehicle to monitor andprovide oversight of the vehicle's operation. The assist driver mayexperience passive fatigue, e.g., fatigue due to under-stimulation orunderload, as autonomous vehicles will typically require less or perhapsno interaction on the part of the assist driver for primary tasksassociated with vehicle operation, e.g., monitoring the SDS. As such,the risk of producing passive fatigue, as well as other phenomenainducing inattention to the automation monitoring task such as driverdistraction and over-reliance on the automation, increases. This mayresult in a corresponding reduction in the ability of the assist driverto take over a primary task associated with vehicle operation if needed.

In one instance, one or more computing devices in a vehicle or SDS thatintroduces secondary tasks or so called interactive cognitive tasks(ICTs) in relation to primary task demand (the level of interactivity ofthe assist driver) may provide a dynamic approach in reducing and/oravoiding passive fatigue in autonomous systems. More specifically, oneor more computing devices may monitor and/or receive informationregarding the performance of primary tasks associated with operating aSDS vehicle and secondary tasks performed by a driver with the SDS.Performance of primary tasks and/or secondary tasks may be measured viaa metric such as primary task demand, which provides a measure of thelevel of driver interactivity. The computing device(s) may thensystematically vary secondary tasks in relation to primary task demand.For example, a time threshold or a distance threshold may be associatedwith primary task demand. If the driver has not interacted with the SDSwithin a time threshold or a distance threshold, the computing device(s)may then trigger a secondary task or ICT.

Primary task demand may be based on the following interactivity markers:comment, driving quality assessment, being within a customer trip. Thecomment marker may include a measure of the interactivity time period(e.g., the elapsed time since the assist driver last interacted with theSDS). Driving quality assessment may be a measure of the importance of acomment from the assist driver. Being within a customer trip is ameasure of requests for use of the self-driving system for a passenger.Other indicators of task demand may include driving conditions such asspeed, road complexity, and/or proximity to cars and pedestrians; thisand other aspects of environmental complexity may also be used ininferring primary task demand. Secondary tasks or ICTs in generalcomprise activities that may engage a driver's motor skills or cognitiveskills. Examples of ICTs include requiring the driver to press aspecific button (e.g., volume up) in response to a voice request. Inthis example, the pressing of the button engages a motor skill, whileunderstanding the voice request engages a cognitive skill. Other ICTsmay include any other appropriate vehicle control, e.g., turning wiperblades on/off, tuning the radio to a specific radio station, etc.

As a working example, a time threshold may be set to 20 minutes (or moreor less) since last driver interaction, or 10 miles (or more or less)since last driver interaction. The computing device may periodicallypoll the SDS to determine driver interaction and receive feedback onprimary task demand. If the driver has not interacted within 20 minutesor 10 miles, then the computing device or SDS may cause an ICT to betriggered. An ICT may comprise a voice request that prompts the driverto complete a task that requires pushing or, more generally, movement ofa specific physical or on-screen button. If the driver does not respondto the ICT within a predetermined time limit (e.g., 10 seconds) orresponds incorrectly, then the computing device or, more generally, SDSmay issue another or follow-up ICT. If no response or an incorrectresponse is received to the follow-up ICT, the SDS or computing devicemay initiate additional action.

Such additional action may include alerting a dispatcher so that thedispatcher may reach out to the driver or activate a video monitoringsystem to determine the state of driver alertness. Other actions mayinclude sounding a distinctive alarm intended to awake even a sleepingdriver, playing loud music or vibrating the driver's seat. Note too thatthe ICT triggering criteria may be adjusted depending on a driver's ICTperformance history, such that drivers with recent errors/misses orslower response times may receive more frequent ICTs. The dispatcher orSDS may also suggest countermeasures that promote driver alertness suchas, for example, taking a break, stretching, etc. When ICT performanceindicates that fatigue is high, more aggressive countermeasures such aspulling the vehicle over may be initiated.

In general, this technology allows for an increase in secondary taskdemand to be inversely proportional to primary task demand. For example,if the vehicle's operation requires more vigilant attention and thusprimary task demand is relatively higher (e.g., driving in a busyintersection), secondary task demand and the frequency of ICTs may bedecreased to make it easier for the driver to focus on the primary task.Conversely, if primary task demand is low (e.g., driving on a desertedhighway), the frequency and complexity of ICTs may be increased.Increasing complexity may comprise requiring more complicated motorskills to complete a voice request (e.g., move a button in a firstdirection and the same or a different button in a second direction).Because primary task demand is an input in determining whether toinitiate secondary tasks or ICTs, the system may avoid initiatingsecondary tasks or ICTs when the driver is engaged in primary tasks.

While the examples above provide a fixed time and/or distance as athreshold, the threshold may also vary in complexity and frequency basedon the operating conditions of the vehicle or a profile of a driver. Forexample, if the SDS or computing device determines that the vehicle hastraversed the distance threshold in an usually fast time (e.g., 10 milesin under 8 minutes correlates to a speed of 80 m.p.h.), that may beconsidered an indication that the vehicle is travelling at higher speedand therefore ICTs may need to be issued more frequently depending onthe perceived attentiveness level of the assist driver. In addition, ifthe driver's profile indicates that the driver gets bored more quicklyon highway drives, the frequency and complexity of ICTs may be increasedto keep such a driver more fully engaged. The complexity and frequencymay also be varied based on driver experience. For example, for a newdriver, ICTs may be triggered more frequently during training or ascompared to a more experienced driver. ICT frequency and complexity maybe varied also by time of day (e.g., fatigue may vary with the time ofday), proximity to meal times or proximity to breaks.

The complexity of a driving context may also serve as input indetermining the timing or complexity of ICTs. For example, theoccurrence of ICTs may be timed so that they occur after, but notduring, driving situations or context where the assist driver should beengaged with the operation of the vehicle. For example, the occurrenceof an ICT may be timed to occur after, but not while, the car istraversing an intersection. Or an ICT might occur while a car is stoppedat a red light. In addition, as the route of an autonomous vehicle isknown, ICTs could also be triggered prior to the occurrence of complexdriving scenarios, thereby having initiation of one or more ICTs occurprior to higher complexity driving contexts. Other inputs may includemap data, road geometry and objects around the vehicle. In this way, anICT may help boost vigilance prior to higher complexity drivingcontexts.

In another aspect, the technology relates to passively detecting driverfatigue of an assist driver in a vehicle configured to operate in afully or partially autonomous mode, or configured for manual operation.Gaze patterns may serve as a reliable indicator of cognitive states,such as for example distractions. Gaze patterns may also serve as areliable indicator of fatigue. For instance, a fatigued driver typicallydoes not vigilantly monitor the road but rather gazes in one direction,e.g., straight ahead. An autonomous vehicle includes sensors and systemsthat can detect objects, etc., surrounding the vehicle and derive sceneunderstanding with respect to the vehicle, and may be able to detect thedirection of the driver's gaze. As such, one or more computing devicesof the vehicle may determine one or more expected gaze direction,including an expected or normative gaze direction, pattern ofdirections, group of directions or sequence of directions (collectivelyreferred to herein as “gaze direction or patterns”), for a driver basedon the derived scene understanding. If the driver is not looking in anexpected gaze direction or pattern on one or more occasions, such as ifthe driver's gaze patterns are not well correlated with an expected gazepattern, this may serve as an indicator of passive fatigue.

For example, if a scene understanding derived from the vehicle'sperception system indicates that based on naturally occurring stimulioutside the vehicle a driver should be looking in a particulardirection, the actual gaze direction or pattern of the driver may thenbe compared to that particular direction or pattern to provide a measureof the driver's attentiveness and thus passive fatigue. As a furtherexample, if the driver is making a right turn, the driver would beexpected to be mainly looking to the left for oncoming traffic. However,if the detected gaze direction or pattern of the driver indicates thatthe driver is looking straight ahead that may signal that the driver isno longer sufficiently engaged in the driving task (e.g., due to lowvigilance or fatigue). As the system can tell when and where drivers areexpected to be looking, it allows for dynamically assessing a driver'svigilance and/or fatigue passively. The actual gaze direction (orpattern of gaze direction) may be detected through use of cameras withinthe compartment of the vehicle.

Different models may be used to determine and/or allocate where a drivershould be looking given the visual stimuli in the field of view. Suchnormative models of visual scanning may be based on recorded attentionallocation behavior of human drivers (using fatigue data as well asattentive driving data) but may also be derived from driving environmentstatistics. The models may use machine learning, or other statisticaltechniques, to learn the statistical properties of such data in order todetermine the appropriate gaze direction or pattern of a driver, e.g.,an assist driver, in a given driving scenario. A normative scanningmodel based on human data may be established by training the model toreproduce the visual scanning patterns associated with attentive driversin similar scenarios. Alternatively, the normative scanning model may beestablished based on environmental statistics according to the followingprinciples: The driving situation will typically comprise a planned pathand the surrounding objects and road infrastructure elements ofrelevance for the planned path. The statistical properties may include,for example, locations at which other objects or road infrastructureelements will typically appear relative to the planned path, thefrequency of those locations, the speed at which the object willapproach those locations and/or the behavior of such objects as theyapproach those locations, etc. Such statistical properties may bemodeled based on data collected while operating a fleet of autonomousvehicles as is described below. An expected scanning model may thereforebe developed for a given situation and used as a benchmark against whicha measured visual scanning pattern of actual drivers may be evaluated.In this way, deviations between the expected gaze direction or patternof a driver based on the normative scanning model may be compared to thedriver's actual gaze direction or pattern as measured by a video, stillcamera, or other gaze detection system in the vehicle to determinewhether the driver is properly allocating his attention or gazedirection. Other models (e.g., machine learning models) may be used toprovide similar input regarding expected or normative gaze direction orpatterns for comparison with actual gaze direction and patterns.

The above features provide for dynamically detecting, monitoring andmanagement of passive driver fatigue in SDS or autonomous vehicles. Thesystem allows for monitoring of driver fatigue state by detecting theactual gaze direction and comparing it to where the driver should belooking, e.g., an expected or normative gaze direction or pattern. Adeviation between the actual gaze direction or pattern and expected ornormative gaze direction or pattern may then be used as an indicatorthat the driver may be fatigued and to trigger an increase in thecognitive load on the driver. The system also allows for monitoring andassessment of driver state using driver interactivity patterns (e.g.,engaged versus not engaged) and then using that assessment to determinethe need for increasing the cognitive load of the driver. The dynamicnature of the technology also makes it safer as secondary task demandsare avoided when drivers are engaged (e.g., during periods when primarytask demand is relatively high). The technology is dynamic not only inthe manner which ICTs or secondary tasks are initiated, but also in thatthe system allows the complexity and frequency of ICTs to dynamicallyadjust according to primary task demand and/or vehicle operatingconditions. The monitoring aspect of the system may be used to morereliably determine the fatigue state of the driver, which may then beused as trigger for ICTs. Thus, the monitoring system and managementsystem (e.g., issuing tasks) may be used collaboratively to complementeach other.

Example Systems

As shown in FIG. 1, a vehicle 100 in accordance with one aspect of thedisclosure includes various components. While certain aspects of thedisclosure are particularly useful in connection with specific types ofvehicles, the vehicle may be any type of vehicle including, but notlimited to, cars, trucks, motorcycles, buses, recreational vehicles,etc. The vehicle may have one or more computing devices, such ascomputing device 110 containing one or more processors 120, memory 130and other components typically present in general purpose computingdevices.

The memory 130 stores information accessible by the one or moreprocessors 120, including instructions 132 and data 134 that may beexecuted or otherwise used by the processor 120. The memory 130 may beof any type capable of storing information accessible by the processor,including a computing device-readable medium, or other medium thatstores data that may be read with the aid of an electronic device, suchas a hard-drive, memory card, ROM, RAM, DVD or other optical disks, aswell as other write-capable and read-only memories. Systems and methodsmay include different combinations of the foregoing, whereby differentportions of the instructions and data are stored on different types ofmedia.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions” and “programs” may be used interchangeably herein.The instructions may be stored in object code format for directprocessing by the processor, or in any other computing device languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. As an example, data 134 of memory130 may store predefined scenarios. A given scenario may identify a setof scenario requirements including a type of object, a range oflocations of the object relative to the vehicle, as well as otherfactors such as whether the autonomous vehicle is able to maneuveraround the object, whether the object is using a turn signal, thecondition of a traffic light relevant to the current location of theobject, whether the object is approaching a stop sign, etc. Therequirements may include discrete values, such as “right turn signal ison” or “in a right turn only lane”, or ranges of values such as “havingan heading that is oriented at an angle that is 20 to 60 degrees offsetfrom a current path of vehicle 100.” In some examples, the predeterminedscenarios may include similar information for multiple objects.

The one or more processor 120 may be any conventional processors, suchas commercially available CPUs. Alternatively, the one or moreprocessors may be a dedicated device such as an ASIC or otherhardware-based processor. Although FIG. 1 functionally illustrates theprocessor, memory, and other elements of computing device 110 as beingwithin the same block, it will be understood by those of ordinary skillin the art that the processor, computing device, or memory may actuallyinclude multiple processors, computing devices, or memories that may ormay not be stored within the same physical housing. As an example,internal electronic display 152 may be controlled by a dedicatedcomputing device having its own processor or central processing unit(CPU), memory, etc. which may interface with the computing device 110via a high-bandwidth or other network connection. In some examples, thiscomputing device may be a user interface computing device which cancommunicate with a user's client device. Similarly, the memory may be ahard drive or other storage media located in a housing different fromthat of computing device 110. Accordingly, references to a processor orcomputing device will be understood to include references to acollection of processors or computing devices or memories that may ormay not operate in parallel.

Computing device 110 may include all of the components normally used inconnection with a computing device such as the processor and memorydescribed above as well as a user input 150 (e.g., a mouse, keyboard,touch screen and/or microphone) and various electronic displays (e.g., amonitor having a screen or any other electrical device that is operableto display information). In this example, the vehicle includes aninternal electronic display 152 as well as one or more speakers 154 toprovide information or audiovisual experiences. In this regard, internalelectronic display 152 may be located within a cabin of vehicle 100 andmay be used by computing device 110 to provide information to passengerswithin the vehicle 100. The vehicle may also include one or morewireless network connections 156 to facilitate communicates with devicesremote from the vehicle and/or between various systems of the vehicle.

In one example, computing device 110 may be an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system may be capable of communicating with various componentsand systems of the vehicle, for instance, wirelessly (via wirelessnetwork connections 156) and/or a wired connection (such as a controllerarea network (CAN) bus or other communication bus). For example,computing device 110 may be in communication with various systems ofvehicle 100, such as deceleration system 160 (for controlling braking ofthe vehicle), acceleration system 162 (for controlling acceleration ofthe vehicle), steering system 164 (for controlling the orientation ofthe wheels and direction of the vehicle), signaling system 166 (forcontrolling turn signals), navigation system 168 (for navigating thevehicle to a location or around objects), positioning system 170 (fordetermining the position of the vehicle), perception system 172 (fordetecting objects in the vehicle's environment), and power system 174(for example, a battery and/or gas or diesel powered engine) in order tocontrol the movement, speed, etc. of vehicle 100 in accordance with theinstructions 132 of memory 130 in an autonomous driving mode which doesnot require or need continuous or periodic input from a passenger of thevehicle. In addition to these systems, the computing device may be incommunication with an interactive cognitive task (ICT) system 178 (fordetecting and tracking driver fatigue, as well as cognitive and othertask oriented interactions with the vehicle 100, the time or distancebetween such interactions, etc.). The ICT system may use signals from avideo or still camera, buttons, switches, dials or other actuators, andother components within the interior of the car to perform its function.Although these systems are shown as external to computing device 110, inactuality, these systems may also be incorporated into computing device110, as an autonomous driving computing system for controlling vehicle100. In addition or alternatively, each of these systems may include oneor more computing devices having processors and memory, configured thesame as or similarly to processors 120 and memory 130 of computingdevices 110 in order to enable the functionalities of these systems asdescribed here.

The computing device 110 may control the direction and speed of thevehicle by controlling various components. By way of example, computingdevice 110 may navigate the vehicle to a destination location completelyautonomously using data from the map information and navigation system168. Computing devices 110 may use the positioning system 170 todetermine the vehicle's location and perception system 172 to detect andrespond to objects when needed to reach the location safely. In order todo so, computing devices 110 may cause the vehicle to accelerate (e.g.,by increasing fuel or other energy provided to the engine byacceleration system 162), decelerate (e.g., by decreasing the fuelsupplied to the engine, changing gears, and/or by applying brakes bydeceleration system 160), change direction (e.g., by turning the frontor rear wheels of vehicle 100 by steering system 164), and signal suchchanges (e.g., by lighting turn signals of signaling system 166). Thus,the acceleration system 162 and deceleration system 160 may be a part ofa drivetrain that includes various components between an engine of thevehicle and the wheels of the vehicle. Again, by controlling thesesystems, computing devices 110 may also control the drivetrain of thevehicle in order to maneuver the vehicle autonomously.

As an example, computing device 110 may interact with decelerationsystem 160 and acceleration system 162 in order to control the speed ofthe vehicle. Similarly, steering system 164 may be used by computingdevice 110 in order to control the direction of vehicle 100. Forexample, if vehicle 100 configured for use on a road, such as a car ortruck, the steering system may include components to control the angleof wheels to turn the vehicle. Signaling system 166 may be used bycomputing device 110 in order to signal the vehicle's intent to otherdrivers or vehicles, for example, by lighting turn signals or brakelights when needed.

Navigation system 168 may be used by computing device 110 in order todetermine and follow a route to a location. In this regard, thenavigation system 168 and/or data 134 may store map information, e.g.,highly detailed maps that computing devices 110 can use to navigate orcontrol the vehicle. As an example, these maps may identify the shapeand elevation of roadways, lane markers, intersections, crosswalks,speed limits, traffic signal lights, buildings, signs, real time orhistorical traffic information, vegetation, or other such objects andinformation. The lane markers may include features such as solid orbroken double or single lane lines, solid or broken lane lines,reflectors, etc. A given lane may be associated with left and right lanelines or other lane markers that define the boundary of the lane. Thus,most lanes may be bounded by a left edge of one lane line and a rightedge of another lane line. As noted above, the map information may storeknown traffic or congestion information and/or and transit schedules(train, bus, etc.) from a particular pickup location at similar times inthe past. This information may even be updated in real time byinformation received by the computing devices 110.

As an example, the detailed map information may include one or moreroadgraphs or graph networks of information such as roads, lanes,intersections, and the connections between these features. Each featuremay be stored as graph data and may be associated with information suchas a geographic location and whether or not it is linked to otherrelated features, for example, a stop sign may be linked to a road andan intersection, etc. In some examples, the associated data may includegrid-based indices of a roadgraph to allow for efficient lookup ofcertain roadgraph features.

The perception system 172 also includes one or more components fordetecting objects external to the vehicle such as other vehicles,obstacles in the roadway, traffic signals, signs, trees, etc. Forexample, the perception system 172 may include one or more LIDARsensors, sonar or other acoustical devices, radar units, cameras (e.g.,video or still, visible light or infrared) and/or any other detectiondevices that record data which may be processed by computing devices110. The sensors of the perception system may detect objects and theircharacteristics such as location, orientation, size, shape, type (forinstance, vehicle, pedestrian, bicyclist, etc.), heading, speed,acceleration, rate of change of acceleration, deceleration, rate ofchange of deceleration, etc. The raw data from the sensors and/or theaforementioned characteristics can be quantified or arranged into adescriptive function, vector, and or bounding box and sent for furtherprocessing to the computing devices 110 periodically and continuously asit is generated by the perception system 172.

As discussed in further detail below, computing devices 110 may use thepositioning system 170 to determine the vehicle's location andperception system 172 to detect and respond to objects when needed toreach the location safely.

For instance, FIG. 2 is an example external view of vehicle 100. In thisexample, roof-top housing 210 and dome housing 212 may include a LIDARsensor as well as various cameras and radar units. In addition, housing220 located at the front end of vehicle 100 and housings 230, 232 on thedriver's and passenger's sides of the vehicle may each store a LIDARsensor. For example, housing 230 is located in front of driver door 250.Vehicle 100 also includes housings 240, 242 for radar units and/orcameras also located on the roof of vehicle 100. Additional radar unitsand cameras (not shown) may be located at the front and rear ends ofvehicle 100 and/or on other positions along the roof or roof-top housing210. Vehicle 100 also includes many features of a typical passengervehicle such as doors 250, 252, wheels 260, 262, etc.

FIGS. 3A and 3B are exemplary internal views or configurations ofvehicle 100. As shown in FIG. 3A, the interior 300A includes an area 306that includes one or more buttons, e.g., buttons L1, L2, L3 and L4. Thebuttons L1, L2, L3 and L4 may comprise push buttons or toggle buttons.These buttons may be specific buttons set aside for responding to audiocomments initiated by ICT task system. They may, however, also comprisebuttons associated with other car functionality. In addition to thesebuttons, area 308 includes a set of buttons that may be used to operatefunctions associated with the audio system of the car, including volumeadjustment, audio mode adjustment (e.g., switch audio source fromsatellite radio to music player), tuning radio stations, etc., inaddition or alternatively to controlling the audio system. In otherexamples, buttons in area 308 may control features associated withdisplay area 310, which contains a speedometer, tachometer, etc. Forexample, the buttons in area 308 may allow for switching between displaymodes in display area 301. Buttons in area 306 (or buttons 306) andbuttons in area 308 (or buttons 308) are shown as being located on oradjacent to steering wheel 312, but may be positioned anywhere withinconvenient and safe reach of an assist driver sitting facing thesteering wheel 312 while sitting in the driver's seat.

The interior 300A also includes speakers 314, which can provide music,phone calls, audio requests initiated by computing device 110 or by orthrough ICT system 172 or other sounds. A screen 318 may also beprovided (e.g., a touch sensitive screen). A windshield wiper controlarm 320 and a headlight/indicator control arm 322 are also providedproximate steering wheel 312. A video or still camera 326 is alsoprovided so that at least an assist driver riding in the interior of thevehicle may be viewed. Braking and acceleration pedals 328 are alsoprovided and may be used, e.g., to override braking or accelerationsettings when the car is operating autonomously or may be used tooperate the car in a manual driving mode. Likewise, steering wheel 312may be engaged by the assist driver when necessary for safe operation ofvehicle 100.

FIG. 3B shows another example interior view or configuration 300B ofvehicle 100. Here, the interior 300B includes speakers 314, video camera326 and brake and acceleration pedals 328, which have similarfunctionality to those discussed in relation to interior 300A. Interior300B also includes area 336, which as shown includes various buttons andknobs that may be used to select features and/or functions associatedwith vehicle 100. For example, area 336 may include buttons 338 ₁ and338 ₂. These buttons may comprise to buttons L1, L2 discussed above.Other buttons, such as L3, L4 are not shown but may also be provided inarea 336. Area 336 may also include various knobs or other actuators3391, 3392 and 3393. Knobs 339 may control functions such as selectionof radio stations, music channels, volume control as well as otherfunctions associated with operation of vehicle 100. User interactionwith knobs 339, as well as buttons 338, may be displayed via screens342, 344. Screens 342, 344 may also comprise touch sensitive screensthrough which various function associated with the vehicle may becontroller, e.g., music selection, volume control, L button operation,windshield wiper operation, etc.

Computing device 110 of vehicle 100 may also receive or transferinformation to and from other computing devices, such as those computingdevices that are a part of the transportation service as well as othercomputing devices, such as those of other vehicles. FIGS. 4 and 5 arepictorial and functional diagrams, respectively, of an example system400 that includes a plurality of computing devices 410, 420, 430, 440and a storage system 450 connected via a network 460. System 400 alsoincludes vehicle 100, and vehicles 100A, 100B which may be configuredthe same as or similarly to vehicle 100. Although only a few vehiclesand computing devices are depicted for simplicity, a typical system mayinclude significantly more.

As shown in FIG. 5, each of computing devices 410, 420, 430, 440 mayinclude one or more processors, memory, data and instructions. Suchprocessors, memories, data and instructions may be configured similarlyto one or more processors 120, memory 130, data 134, and instructions132 of computing device 110.

The network 460, and intervening nodes, may include variousconfigurations and protocols including short range communicationprotocols such as Bluetooth™, Bluetooth™ LE, the Internet, World WideWeb, intranets, virtual private networks, wide area networks, localnetworks, private networks using communication protocols proprietary toone or more companies, Ethernet, WiFi and HTTP, and various combinationsof the foregoing. Such communication may be facilitated by any devicecapable of transmitting data to and from other computing devices, suchas modems and wireless interfaces.

In one example, one or more computing devices 410 may include one ormore server computing devices having a plurality of computing devices,e.g., a load balanced server farm, that exchange information withdifferent nodes of a network for the purpose of receiving, processingand transmitting the data to and from other computing devices. Forinstance, one or more computing devices 410 may include one or moreserver computing devices that are capable of communicating withcomputing device 110 of vehicle 100 or a similar computing device ofvehicle 100A, 100B as well as computing devices 420, 430, 440 via thenetwork 460. For example, vehicles 100, 100A, 100B may be a part of afleet of vehicles that can send and receive information from the servercomputing devices 410. In this regard, the server computing devices 410together may function as a fleet management system, each of the servercomputing devices functioning to perform one or more roles of the fleetmanagement system. In addition, the server computing devices of thefleet management system may use network 460 to transmit and presentinformation to a user, such as user 422, 432, 442 on a display, such asdisplays 424, 434, 444 of computing devices 420, 430, 440. In thisregard, computing devices 420, 430, 440 may be considered clientcomputing devices.

As shown in FIG. 4, each client computing device 420, 430, 440 may be apersonal computing device intended for use by a user 422, 432, 442, andhave all of the components normally used in connection with a personalcomputing device including a one or more processors (e.g., a centralprocessing unit (CPU)), memory (e.g., RAM and internal hard drives)storing data and instructions, a display such as displays 424, 434, 444(e.g., a monitor having a screen, a touch-screen, a projector, atelevision, or other device that is operable to display information),speakers, and user input devices 426, 436, 446 (e.g., a mouse, keyboard,touchscreen or microphone). The client computing devices may alsoinclude a camera for recording video streams, speakers, a networkinterface device, and all of the components used for connecting theseelements to one another.

Client computing device 440 may also be a workstation for a customerservice representative. In this regard, user 422 may be a customerservice representative or dispatcher who can communicate with passengersor assist drivers of the vehicles 100, 100A, and 100B when connected bya server computing device as discussed further below. In addition, theclient computing device 440 may enable the user 422 to accessinformation about the vehicles of the fleet stored in the storage system450, for instance, by communicating with the server computing devices410 of fleet management system over network 460. Again, although only asingle customer service work station is depicted in FIGS. 4 and 5, thesystem may actually include tens or hundreds of such workstations andcustomer service representatives.

Although the client computing devices 420, 430, and 440 may eachcomprise a full-sized personal computing device, they may alternativelycomprise mobile computing devices capable of wirelessly exchanging datawith a server computing device (such as the server computing devices410) over a network such as the Internet. By way of example only, clientcomputing device 420 may be a mobile phone or a device such as awireless-enabled PDA, a tablet PC, a wearable computing device orsystem, or a netbook that is capable of obtaining information via theInternet or other networks. In another example, client computing device430 may be a wearable computing system, shown as a smartwatch as shownin FIG. 5. As an example the user may input information using a smallkeyboard, a keypad, microphone, using visual signals with a camera, or atouch screen.

As with memory 130, storage system 450 can be of any type ofcomputerized storage capable of storing information accessible by theserver computing devices 410, such as a hard-drive, memory card, ROM,RAM, DVD, CD-ROM, write-capable, and read-only memories. In addition,storage system 450 may include a distributed storage system where datais stored on a plurality of different storage devices which may bephysically located at the same or different geographic locations.Storage system 450 may be connected to the computing devices via thenetwork 460 as shown in FIGS. 4 and 5, and/or may be directly connectedto or incorporated into any of the computing devices 110, 410, 420, 430,440, etc.

Storage system 450 may store various types of information as describedin more detail below. This information may be retrieved or otherwiseaccessed by one or more server computing devices, such as those of thefleet management system, and/or one or more client computing device,such as the client computing device 440, in order to perform some or allof the features described herein.

In addition, the information of storage system 450 may store informationabout the status and characteristics of each vehicle of the fleet, asdiscussed above, as well as the map information discussed above. As thevehicles of the fleet drive around, they may constantly and/orperiodically broadcast to one or more of the server computing devices410 of the fleet management system their status. This may include, forexample, whether the vehicle is currently on a trip (e.g., istransporting passengers and/or cargo), a current destination of thevehicle and/or one or more additional future destinations for thevehicle (as discussed further below), whether the vehicle has anymaintenance needs, etc. As an example, maintenance needs may includewhether the vehicle needs cooling or shade, refueling or charging,cleaning, periodic inspection (after so many hours, trips, or miles inservice), recalibration of sensors (to address faults or periodicmaintenance), or other maintenance. The one or more server computingdevices 410 may store his information in storage system 450.

Storage system 450 may also store information about assist drivers. Suchinformation may include driver profile (age, language spoken, experiencelevel, contact information, etc.), driver shift information (e.g., whenthe driver shift started/ends, time since last break, etc.). Protectionswill be put in place to ensure that user privacy remains protected. Forexample, assist drivers may be required to specifically authorizestorage of certain information about themselves. Additional protectionsinclude anonymization of personally identifiable information,aggregation of data, filtering of personal information, encryption,hashing or filtering of personal information to remove personalattributes, time limitations on storage of information, or limitationson data use or sharing.

The storage system 450 may also store information about requests forcustomer service. This information may include, for instance, a type ofthe request, how the request was originated, a vehicle of the fleetassociated with the request, when the request was generated (atimestamp), an associated priority level, whether the request isassigned to a queue (and if so, which if there is more than one queue),whether the request is completed, etc. In addition, the storage system450 may also be used to store the aforementioned queues and/or theinformation about the queues as discussed further below.

The storage system 450 may also store information about customer servicerepresentatives or dispatchers, such as user 442. This may includeinformation for each customer service representative, such as the numberof requests serviced, passenger (or users) of the server which thecustomer service representative has communicated with previously, aswell as characteristics of that customer service representative, such aslevel of experience, ranking or ratings, skill set, certification,language spoken, or other such characteristics. As with assist drivers,personal information about customer service representatives ordispatchers may also be subject to heightened protections.

In order to provide transportation services to users, the information ofstorage system 450 may include user account information such ascredentials (e.g., identifiers such as a username and password as in thecase of a traditional single-factor authentication as well as othertypes of credentials typically used in multi-factor authentications suchas random identifiers, biometrics, etc.) that can be used to identify auser to the fleet management system. The user account information mayalso include personal information such as the user's name, contactinformation, identifying information of the user's client computingdevice (or devices if multiple devices are used with the same useraccount), one or more unique signals for the user, whether the user haschosen to opt for one or more different pre-determined levels of serviceor otherwise has a need for accessibility services, as well as otheruser preference or settings data.

As with memory 130, storage system 450 can be of any type of computerstorage capable of storing information accessible by the servercomputing devices 410, such as a hard-drive, memory card, ROM, RAM, DVD,CD-ROM, write-capable, and read-only memories. In addition, storagesystem 450 may include a distributed storage system where data is storedon a plurality of different storage devices which may be physicallylocated at the same or different geographic locations. Storage system450 may be connected to the computing devices via the network 460 asshown in FIGS. 4 and 5 and/or may be directly connected to orincorporated into any of the computing devices 110, 410, 420, 430, 440,etc.

Example Methods

In addition to the operations described above and illustrated in thefigures, various operations will now be described. It should beunderstood that the following operations do not have to be performed inthe precise order described below. Rather, various steps can be handledin a different order or simultaneously, and steps may also be added oromitted. For instance, FIG. 6 is an example flow diagram 600 that may beperformed by one or more processors, such as one or more processors ofcomputing devices 110 in vehicles 100, 100A, or 100B, in order tomonitor and mitigate driver fatigue in one or more of vehicles 100, 100Aand/or 100B. The flow diagram 600 may be performed as a process onserver 410. In such an implementation each car could provide data toserver 401 so that the server would then perform the process of flowdiagram 600 to administer the ICT in combination with otherconsideration such as for example time of day, driver experience, timesince last ICT or as further described throughout.

As shown in block 610 of FIG. 6, computing device 110 monitors driverinteractions with the vehicle or SDS. These interactions includeperformance of primary tasks associated with operation of the vehicle orsecondary or ICT tasks initiated by the computing device 110 or the SDSto stimulate interaction by the driver. The primary tasks may include adriver providing a comment relating to the operation of the vehicle, adriving quality assessment or the vehicle being use to carry a passengerto a destination. Secondary tasks or ICTs generally include tasks thatengage a driver's motor skills or cognitive skills. Motor skillscomprise acts that involve muscle use, such as for example, pressing abutton, toggling a rocker switch, limb movement, etc. Cognitive skillsare based on the use of the brain and may require one or more ofthinking, reading, remembering, reasoning or paying attention. As anexample, cognitive skills are invoked in response to a request toperform an act such as pressing a specific button as it involvesthinking about the request, remembering it and focusing one's attentionlong enough to complete the requested act.

The computing device may measure the performance of primary tasks and/orsecondary tasks using a primary task demand metric. The primary taskdemand metric can comprise a measurement of distance travelled and/ortime travelled since a primary task or secondary task was last performedby an assist driver. The distance measurement may be accumulated by acounter linked to the vehicle's system that tracks distance travelled.The counter may then provide the distance travelled to the computingdevice 110. Alternatively, the vehicle system may provide the signals tocomputing device 110, which may itself implement a distance counter.Time travelled can be tracked using a timer implemented via computingdevice 110 or having a timer that feeds its output to computing device110. Computing device 110 may receive counter or timer information byperiodically polling devices that perform those functions or via aconnection that provides such information in real time. The primary taskdemand metric may comprise a measurement of both primary taskperformance and secondary task performance. The metric, however, mayonly comprise a measure of either primary tasks or secondary tasks. Inaddition, two separate demand metrics may be tracked by the computingdevice, e.g., a secondary task only metric or a primary task onlymetric.

As shown at block 620 of FIG. 6, the primary task demand metric measuredis compared to a threshold. In an example where the primary task demandmetric includes a measurement of both distance and time, in one scenariothe distance threshold may be set to 10 miles and the time threshold maybe set to 20 minutes. The distance threshold may incorporate some levelof randomness and vary for example by +/−3 miles (e.g., up to 30%variance). For example, if the distance threshold is initially set to 10miles, when a task is performed it may then be reset to 7 miles, when asubsequent task is received after the reset the threshold may then beset back 10 miles, then reset thereafter to 13 miles, etc. Introducingrandomness aids in avoiding training the driver to circumvent the systemby knowing the exact time period or distance that tasks need to becompleted.

The time threshold may be a hard time limit to account for situationswhere, for example, it may take too long of a time to traverse a givendistance due to traffic patterns such that the distance threshold mightprove to not be a useful threshold. In addition, a measure of randomnessmay be introduced with respect to the timing threshold similar to thatdiscussed above in relation to the distance threshold. For example,there may be up to a 30% (or more) variance in the timing threshold

If the distance or time travelled since either a primary task orsecondary task was performed meets or exceeds the respective threshold,as shown at block 630 of FIG. 6, the computing device 110 initiates asecondary task or ICT for completion by the driver. For example, thecomputing device 110 may initiate a request through speakers 314 for thedriver to “please press the L2 button.” The driver is then given apredetermined time, e.g., 10 seconds, to complete the requested action.Depending on the driver's response additional ICTs may be issued,countermeasures may be invoked if it is determined that the driver isfatigued, or the system may reset the counters and start the monitoringprocess at block 610.

As discussed above, the primary task demand metric includes as acomponent monitoring of the performance of primary task. As such,initiation of ICTs or secondary tasks will vary dynamically in relationto the frequency at which primary tasks are performed. For example,where the driving context requires regular interaction with the vehicleor SDS within time or distance threshold limits (e.g., the demand forprimary task performance is relatively high), ICTs will not be issued.In contrast, when the demand for the performance of primary tasks islow, ICTs will be issued more regularly. Thus, secondary task demandvaries dynamically in relation to primary task demand. In this regard,in an aspect of the disclosure secondary task demand increases ininverse proportion to primary task demand. This allows for moreeffective counteraction of different levels of fatigue. In addition, asystem implemented in accordance with this disclosure maybe safer asprimary task demands are not initiated or issued when drivers are highlyengaged (e.g., primary task demand is high).

The system may also dynamically manage and mitigate passive fatigue byadapting the complexity of ICT or secondary tasks or frequency at whichtasks are initiated based on the driving context and other factors. Forexample, factors such scene understanding (e.g., the location andmovement of external objects within a vicinity of the vehicle), theterrain or route being travelled, the driver profile, weatherconditions, speed of the vehicle, time of day, traffic conditions, etc.In effect, these factors may impact the driver context and therebyrequire different levels of vigilance or attention by the driver. Forexample, if perception system 172 indicates that the scene in thevicinity of the vehicle include multiple objects such that vehicle in acomplex environment, the timing and complexity of ICTs may be adjustedto accommodate for such an environment. As another example, if vehicleis on relatively long trip, the frequency and complexity of ICTs mayincrease in proportion to distance travelled or length of time on thetrip. As another example, nighttime driving may also require that thefrequency and complexity of ICTs increase. The foregoing factors may betaken into account in setting the distance and time thresholds discussedabove and set by server 410 or a dispatcher.

Examples of more complex ICTs may include a sequence in which more thanone button is operated, e.g., press L2 once, L3 twice, and L4 threetimes. As another example, an ICT may include a request to tune theradio to a particular station frequency. Other examples includeoperating the windshield wiper, opening a particular window, etc. Asanother example, as the SDS is able to sense the environment, an ICT maycomprise asking a driver to report or describe scenes or objects intheir driving environment. Poor performance on such an ICT may providean indication of poor vigilance due to fatigue or distraction. As longas computing device 110 is able to be receive information indicatingthat the task requested was completed, any feature of the vehicle thatwon't impact the safe operation of the vehicle may be used in an ICT. Inthat regard, any ICT that provides cognitive engagement that mitigatesfatigue may be used.

In another instance, FIGS. 7A and 7B is a further example flow diagram700 with operations that may be performed by one or more processors,such as one or more processors of computing devices 110 in vehicles 100,100A, or 100B in order to monitor and mitigate driver fatigue in one ormore of vehicles 100, 100A and/or 100B. As shown at block 710 of FIG.7A, computing device 110 determines at diamond 710 whether to initiatethe ICT system. In making this determination, the processing device maydetermine whether the vehicle is moving, whether the parking brake isreleased whether there are dual drivers, whether passengers are presentin the interior of the vehicle, whether the vehicle is in manual mode ora partially autonomous mode, or whether there is no assist driver. Forexample, if the vehicle is stationary or not moving processing returnsto diamond 710 via line 715. On the other hand, if the vehicle ismoving, processing proceeds to block 720. Similarly, if the parkingbrake is released, the process proceeds to block 720. Likewise, if theother parameters indicate that the ICT system should be engaged becausethere is a single assist driver in the vehicle, there are no passengersin the vehicle, and that the car is not in manual mode, processingproceeds to block 720. In this regard, where another driver or passengeris in the vehicle, the ICT system may not need to be engaged as theexpectation is that by having another person in the interior of thevehicle the assist driver will most likely be more vigilant andattentive. The ICT system may also shut off when the car nears apassenger pick up location or once the car goes into park at thepassenger pick up location.

With ICT system initiated at block 720, the computing device 110monitors the SDS for driver interaction or receives information from theSDS indicative of driver interaction with the SDS, starts a distancecounter and monitor timer. As discussed above, the interaction maycomprise a primary task or a secondary task/ICT.

As indicated at diamond 726, if appropriate driver interaction isdetected before the distance counter or monitor timer reaches respectivedistance or time thresholds, the process returns to block 720, via line728, where the distance counter and monitor timer are reset. Asdiscussed above, appropriate driver interaction includes eitherperformance of primary task or a secondary task. If no driverinteraction is detected before the distance counter or monitor timerreach their thresholds, then processing proceeds to block 730. As notedabove, the time thresholds will typically be set to a hard time limit.This is to account for situations where it may take a much longer timeto traverse a given distance due to traffic patterns and the distancethreshold may not be as useful.

At block 730, a first ICT or secondary task is initiated by computingdevice 110 and an ICT response timer is started. For example, a drivermay be requested to “press the L2 button.” That driver then has apredetermined time period, e.g., 10 to 30 seconds, to record a responsewith time being measured by the ICT response timer.

If, as indicated at diamond 734, the driver provides a correct responsebefore the ICT response timer times out, then processing returns toblock 720 as indicated via line 739. On the other hand, if the driverprovides an incorrect answer or the ICT response timer times out,processing proceeds to block 750 in FIG. 7B.

At block 750, a second ICT or secondary task is initiated and theresponse timer is reset. The second ICT will typically be different thanthe first ICT. For example, it may request, “please press the L3button.” If, as indicated at diamond 754, the driver provides a correctresponse before the response times out, then the process returns toblock 720 as indicated via line 755. On the other hand, if the driverprovides an incorrect answer or the response timer times out, processingproceeds to block 760.

At block 760, the computing device may initiate communication or cause acommunication to be sent to a dispatcher. The communication may be anemail, text message or a phone call. The dispatcher may then reach outto the driver to inquire the driver's level of alertness and suggestthat they employ countermeasures to enhance alertness, e.g., breaks,stretching, etc. Note also that if the second ICT fails to garner atimely and correct response, the event will be logged at block 768.

While block 760 indicates that a communication will be initiated to adispatcher in the event the second ICT is not timely responded to, notethat the system may loop through the process with a third ICT in someexamples, and in other examples, other countermeasures may be employedprior to, or in addition to, contacting the dispatcher. For example, thevolume of the radio or music player in the vehicle may be increased to alevel that should wake even a sleeping person. Alternatively, thedriver's side window may be opened enough to let in additional outsideair or the horn may be engaged. In essence, any vehicle feature that isaccessible via the SDS that won't affect vehicle operation or compromisethe safety of the driver (or passengers) may be initiated to obtain theattention of the assist driver. If such countermeasures are employed,then additional steps may be added to flow 700 including initiatingadditional ICTs and waiting for a response. Ultimately, however, if noresponse is forthcoming, then a dispatcher would be contacted.

In another instance, example flow diagram 800 as shown in FIG. 8illustrates a further scenario that may be performed by one or moreprocessors, such as one or more processors of computing devices 110 invehicles 100, 100A, or 100B, in order to detect potential driver fatiguein one or more of vehicles 100, 100A and/or 100B. At block 810, anactual gaze direction (or pattern of gaze direction) of the driver ismonitored, using for example the camera 326 of FIG. 3. Camera 326 may befocused on one or both of the driver's eyes such it captures digitalimages of the eye(s). Computing device 110 may then use the digitalimage information to calculate the actual eye gaze direction of thedriver. In parallel, at block 820, the computing device or SDS mayreceive information that about the planned drive path and objects in thevicinity of the vehicle along the drive path. The measured gazedirection and location surrounding objects can then be combined todetermine what object the driver is looking at. Where a driver iswearing sunglasses, emitters may be selected such the wavelength oflight, e.g., infrared, passes through the sunglasses. Otherwise, properplacement of the camera should allow for capture of the driver's gazedirection.

At block 840, information about the planned drive path and objects areused to determine an expected or normative gaze direction or pattern asdefined by a normative scanning model. The normative gaze direction orpattern may be considered an optimal gaze direction in somecircumstances, e.g., where there is sufficient confidence in the model.The normative scanning model may be determined based on a visualscanning model developed from actual human scanning data and/or based onenvironmental statistics. A fundamental role of visual scanning indriving is to reduce uncertainty of a planned or predicted driving path.This uncertainty may be divided into (1) uncertainty regarding drivingpath and (2) uncertainty regarding location and movement of objectsexternal to the vehicle in relation to the drive path. For a givendriving situation, a strategy may be employed for visually scanning theenvironment that works to minimize uncertainty regarding traversing theplanned path. Such strategy may be determined by the statisticalproperties of the driving context (e.g., locations at which otherobjects may appear, their speed, their behavior, etc.). For example, ifa driver is making a left onto two lane road, the driver will look leftand right at least once. Human drivers learn these statisticalproperties over time through experience and, thus, the normativescanning model may be derived based on visual scanning patterns recordedfrom experienced attentive drivers. These statistical properties mayalso be estimated from data that is collected as part of SDS fleetmanagement platform and used to derive an expected visual scanning modelfor a specific driving situation or context, without the use of humandata, based on the principle that optimal scanning will minimizeuncertainty relative to the planned path in the given scenario. Thescanning model is used to derive an expected gaze direction or patterngiven the driving context, given an expected drive path and objectsalong the drive path.

At block 860, the expected gaze direction or pattern may then be used asa benchmark and compared to the actual gaze direction or patterndetected by camera 326. If there is deviation between the actual gazedirection or pattern and the benchmark, then at block 880 the ICT systemmay be initiated as previously discussed to initiate ICT requests to thedriver. A deviation between a driver's actual visual scanning patternand that prescribed by the expected model indicates a reduced ability toproperly allocate attention to support safe driving, which may berelated to (among other things) driver distraction, fatigue orover-reliance on automation (e.g., a test driver not scanning the roadproperly as they started to over-trust the vehicle operating in anautonomous driving mode).

The foregoing disclosure was discussed in the context of a self-drivingsystem. Note, however, that the foregoing methods and processes may beimplemented in non-SDS scenario. For example, the vehicle in FIG. 3A maybe operated in a manual mode while the ICT system is fully engaged. Inthis the method steps of FIGS. 6 through 8 may be implemented andpracticed to mitigate possible driver fatigue as the vehicle is operatedmanually.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements.

1. A method for managing driver inattention, comprising: acquiring, byone or more processors, images of one or both of a driver's eyes todetermine an actual gaze direction of the driver or an actual gazepattern of the driver of a vehicle; determining, by the one or moreprocessors, an expected gaze direction of the driver or an expected gazepattern of the driver based on at least one of a planned route of thevehicle and objects detected in an external environment of the vehicle;comparing, by the one or more processors, (i) the actual gaze directionto the expected gaze direction or (ii) the actual gaze pattern to theexpected gaze pattern; determining, by the one or more processors basedon the comparison, whether the driver's visual scanning of the externalenvironment deviates from the expected gaze direction or the expectedgaze pattern; and taking corrective action upon determining that thedriver's visual scanning of the external environment deviates by anamount that indicates driver inattention.
 2. The method of claim 1,wherein determining whether the driver's visual scanning of the externalenvironment deviates from the expected gaze direction or the expectedgaze pattern comprises using a normative model that evaluates one ormore statistics.
 3. The method of claim 2, wherein the normative modelis based on environmental statistics to determine the expected gazedirection or the expected gaze pattern given a planned driving path. 4.The method of claim 2, wherein the normative model is based on humanvisual behavior statistics to determine the expected gaze direction orthe expected gaze pattern.
 5. The method of claim 2, wherein the one ormore statistics of the normative model include at least one of (i)locations at which other objects or road infrastructure elements mayappear relative to a planned driving path, (ii) a frequency of thelocations, (iii) speeds at which the other objects are expected toapproach the locations, or (iv) behavior of the objects as they approachthe locations.
 6. The method of claim 2, wherein the normative model isassociated with at least one of (i) an uncertainty regarding the planneddriving path, or (ii) an uncertainty regarding a location and movementof objects external to the vehicle in relation to the planned drivingpath.
 7. The method of claim 2, wherein statistical propertiesassociated with the normative model are estimated from data collected bya fleet of vehicles configured to operate in an autonomous driving mode.8. The method of claim 1, wherein acquiring the images comprisesmonitoring one or both of the driver's eyes for eye closure.
 9. Themethod of claim 8, wherein the monitoring includes determining whetherthe driver is wearing sunglasses.
 10. The method of claim 8, wherein themonitoring includes selecting one or more emitters such that a selectedwavelength of light is able to pass through sunglasses worn by thedriver.
 11. The method of claim 1, wherein the driver is a driver of anon-autonomous vehicle or an assist driver monitoring the operations ofan autonomous vehicle.
 12. The method of claim 1, wherein taking thecorrective action includes the one or more processors initiating one ormore features of the vehicle to obtain attention of the driver.
 13. Themethod of claim 1, wherein taking the corrective action includes the oneor more processors initiating an interactive cognitive task with thedriver.
 14. The method of claim 13, wherein the one or more processorsdetermine when to initiate the interactive cognitive task based on atleast one of (i) an operational status of the vehicle, (ii) whetherthere are dual drivers, (iii) whether passengers are present in thevehicle, or (iv) whether the vehicle is in a manual driving mode or apartially autonomous driving mode.
 15. A vehicle comprising: a drivingsystem including a steering subsystem, an acceleration subsystem and adeceleration subsystem to control driving of the vehicle; a perceptionsystem including one or more sensors configured to detect objects in anenvironment external to the vehicle; a positioning system configured todetermine a current position of the vehicle; and a control systemincluding one or more processors, the control system operatively coupledto the driving system, the perception system and the positioning system,the control system being configured to: acquire, from the perceptionsystem, images of one or both of a driver's eyes to determine an actualgaze direction of the driver or an actual gaze pattern of the driver;determine an expected gaze direction of the driver or an expected gazepattern of the driver based on at least one of a planned route of thevehicle and objects detected in an external environment of the vehicle;compare (i) the actual gaze direction to the expected gaze direction or(ii) the actual gaze pattern to the expected gaze pattern; determine,based on the comparison, whether the driver's visual scanning of theexternal environment deviates from the expected gaze direction or theexpected gaze pattern; and take corrective action upon determining thatthe driver's visual scanning of the external environment deviates by anamount that indicates driver inattention.
 16. The vehicle of claim 15,wherein the determination of whether the driver's visual scanning of theexternal environment deviates from the expected gaze direction or theexpected gaze pattern comprises use of a normative model that evaluatesone or more statistics.
 17. The vehicle of claim 16, wherein thenormative model is based on at least one of (i) environmental statisticsto determine the expected gaze direction or the expected gaze patterngiven a planned driving path, or (ii) human visual behavior statisticsto determine the expected gaze direction or the expected gaze pattern.18. The vehicle of claim 15, wherein: the perception system furtherincludes one or more in-vehicle sensors configured to detect informationregarding the driver's eye position; and acquisition of the imagescomprises monitoring, by the one or more in-vehicle sensors, one or bothof the driver's eyes for eye closure.
 19. The vehicle of claim 15,wherein taking the corrective action includes initiation of one or morefeatures of the vehicle to obtain attention of the driver.
 20. Thevehicle of claim 15, wherein taking the corrective action includesinitiation of an interactive cognitive task with the driver.